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Abstract 


This report presents the mechanical verification of a 
simplified model of a rapid Byzantine-fault-tolerant self-stabilizing 
protocol for distributed clock synchronization systems. This 
protocol does not rely on any assumptions about the initial state of 
the system. This protocol tolerates bursts of transient failures, and 
deterministically converges within a time bound that is a linear 
function of the self-stabilization period. A simplified model of the 
protocol is verified using the Symbolic Model Verifier (SMV) 
[SMV]. The system under study consists of 4 nodes, where at most 
one of the nodes is assumed to be Byzantine faulty. The model 
checking effort is focused on verifying correctness of the simplified 
model of the protocol in the presence of a permanent Byzantine 
fault as well as confirmation of claims of determinism and linear 
convergence with respect to the self-stabilization period. Although 
model checking results of the simplified model of the protocol 
confirm the theoretical predictions, these results do not necessarily 
confirm that the protocol solves the general case of this problem. 
Modeling challenges of the protocol and the system are addressed. 
A number of abstractions are utilized in order to reduce the state 
space. Also, additional innovative state space reduction techniques 
are introduced that can be used in future verification efforts 
applied to this and other protocols. 
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1. Introduction 


The concept of self-stabilizing distributed computation was first presented in a classic 
paper by Dijkstra [Dijkstra 1974]. In that paper, he speculated whether it would be possible for a 
set of machines to stabilize their collective behavior in spite of unknown initial conditions and 
distributed control. A fundamental criterion in the design of a robust distributed system is to 
provide the capability of tolerating and potentially recovering from failures that are not 
predictable in advance. Overcoming such failures is most suitably addressed by tolerating 
Byzantine faults [Lamport 1982], There are many algorithms that address permanent faults 
[Srikanth 1987], where the issue of transient failures is either ignored or inadequately addressed. 
There are many efficient Byzantine clock synchronization algorithms that are based on 
assumptions on initial synchrony of the nodes [Srikanth 1987, Welch 1988] or existence of a 
common pulse at the nodes, e.g. the first protocol in [Dolev 2004]. There are many clock 
synchronization algorithms that are based on randomization and, therefore, are non- 
deterministic, e.g. the second protocol in [Dolev 2004]. 

Solving these special cases is insufficient to claim that an algorithm is self-stabilizing. 
The main challenges associated with self-stabilization are the complexity of the design and the 
proof of correctness of the protocol. Another difficulty is achieving an efficient convergence 
time for the proposed self-stabilizing protocol. Typically, verification of a protocol is conducted 
by the composition of a paper-and-pencil proof. Verification of such proofs is another challenge 
associated with self-stabilization, especially as the complexity of the protocol increases. Such 
proofs are error prone. One recent work in this area is the algorithm developed by Daliot et al 
[Daliot 2003] called the Byzantine self-stabilization pulse synchronization (BSS-Pulse-Synch) 
protocol. A flaw in BSS-Pulse-Synch protocol was found and documented in a report by 
Malekpour et al. [Malekpour 2006 A]. Such flaws are harder to pinpoint in the proof argument 
than finding a counterexample via simulation or model checking. 

Another technique sometimes used to verify the correctness of a design is based on 
extensive simulation but it too can miss significant errors when the number of possible states is 
very large. Simulation of specific scenarios requires proper set up of the system for each case. 
As the number of cases to be examined increases, this process becomes impractical. 

Model checking is a method for mechanically verifying finite-state concurrent systems. 
Specifications about the system are expressed as temporal logic formulas, and efficient symbolic 
algorithms are used to traverse the model defined by the system and check if the specification 
holds or not. The verification procedure is an exhaustive search of the state space of the design. 
As a result, model checking is a viable means for mechanically verifying the claims of a 
distributed clock synchronization protocol. Model checking also provides insight into the 
behavior of the system even if it cannot fully explore the entire state space. Therefore, model 
checking is a practical alternative for accessing correctness of a protocol and proving correctness 
of a protocol instance. 

This report presents model checking efforts in support of the claims of a rapid Byzantine- 
fault-tolerant self- stabilizing protocol for distributed clock synchronization systems [Malekpour 
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2006B, 2006C]. In particular, this effort encompasses the verification of correctness of a 
simplified model of the protocol by confirming that a candidate system self-stabilizes from any 
state and tolerates bursts of transient failures in the presence of permanent Byzantine faulty 
nodes. A permanent Byzantine faulty node is a node with arbitrarily malicious behavior. This 
effort, furthermore, includes the verification of claims of determinism and linear convergence of 
the simplified model of the protocol with respect to the self-stabilization period and in the 
presence of permanent Byzantine faulty nodes. Although model checking results of the 
simplified model of the protocol are promising, these results do not necessarily imply that the 
protocol solves the general case of this problem. 



Figure 1 . A 4-node system. 

As shown in Figure 1, the system under study consists of 4 nodes, where 3 of the nodes 
are assumed to be good and one of the nodes is Byzantine faulty. Toward this objective, a 
number of abstractions and reduction techniques are devised to reduce the state space. Also, in 
order to further reduce the state space to a more manageable size, system parameters are reduced 
to their minimal values. The amount of memory needed for the construction of the Binary 
Decision Diagram (BDD) readily reaches the 4GB available after construction of the state space. 
Therefore, model checking of larger and more complex systems poses a greater challenge. 

The following sections describe the model checking efforts in detail. The report begins 
with a description of the protocol followed by a brief history of the model checking effort. 
Modeling specifications and abstractions used in describing a basic case of this protocol are 
described in the following section. The underlying topology and network models are defined, 
followed by the SMV models of the individual parts. The propositions are then enumerated. A 
summary of the model checking results is presented. Additional reduction techniques are also 
introduced, followed by the concluding remarks. 


2. The Protocol 

A distributed system is defined to be self-stabilizing if, from an arbitrary state and in the 
presence of bounded number of Byzantine faults, it is guaranteed to reach a legitimate state in a 
finite amount of time and remain in a legitimate state as long as the number of Byzantine faults 
are within a specific bound. A legitimate state is a state where all good clocks in the system are 
synchronized within a given precision bound. 
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The self-stabilization problem has two facets. First, it is inherently event-driven and, 
second, it is time-driven. Most attempts at solving the self-stabilization problem have focused 
only on the event-driven aspect of this problem. The protocol presented here properly merges 
the time and event driven aspects of this problem in order to self-stabilize the system in a gradual 
and yet timely manner. Furthermore, this protocol is based on the concept of a continual 
vigilance of the state of the system in order to maintain and guarantee its stabilized status, and a 
periodic reaffirmation of nodes by declaring their internal status. Finally, initialization and/or 
reintegration are not treated as special cases. These scenarios are regarded as inherent parts of 
this self-stabilizing protocol. 

The self-stabilization events are captured at a node via a selection function that is based 
on received valid messages from other nodes. When such an event occurs, it is said that a node 
has accepted or an accept event has occurred. In order to achieve self-stabilization, the nodes 
communicate by exchanging two self-stabilization messages labeled Resync and Affirm. The 
Resync message reflects the time-driven aspect of this self-stabilization protocol, while the 
Affirm message reflects the event-driven aspect of it. The Resync message is transmitted when a 
node realizes that the system is no longer stabilized or as a result of a resynchronization timeout. 
The Affirm message is transmitted periodically and at specific intervals primarily in response to a 
legitimate self-stabilization accept event at the node. 

The time difference between interdependent consecutive events is expressed in terms of 
the minimum event-response delay, D, and network imprecision, d. As a result, the approach 
presented here is expressed as a self-stabilization of the system as a function of the expected time 
separation between the consecutive Affirm messages, A aa . To guarantee that a message from a 
good node is received by all other good nodes before a subsequent message is transmitted, Aaa is 
constrained such that Aaa > (D + d). Unless stated otherwise, all time dependent parameters of 
this protocol are measured locally and expressed as functions of Aaa- 

Three fundamental parameters characterize the self-stabilization protocol presented 
here, namely K, D, and d. The number of faulty nodes, F, the number of good nodes, G, and the 
remaining parameters that are subsequently enumerated are derived parameters and are based 
on these three fundamental parameters. Furthermore, except for K, F, G, T A and T R , which are 
integer numbers, other parameters are real numbers. In particular, Aaa is used as a threshold 
value for monitoring of proper timing of incoming and outgoing Affirm messages. The derived 
parameters T A = G - 1 and T R = F + 1 are used as thresholds in conjunction with the Affirm and 
Resync messages, respectively. 

The assessment results of the monitored nodes are utilized by the node in the self- 
stabilization process. The node consists of a state machine and a set of (K-\ ) monitors. The state 
machine has two states, Restore state ( T) and Maintain state (M), that reflect the current state of 
the node in the system as shown in Figure 2, where Resync messages are represented as R and 
Affirm messages are represented as A. 
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R, A A A 



Figure 2. The node state machine. 


2.1. Transitory Conditions 

The transitory conditions enable the node to migrate to the Maintain state and are 
defined as: 

1. The node is in the Restore state, 

2. At least 2 F accept events in as many Aaa intervals have occurred after the node entered 
the Restore state, 

3. No valid Resync messages are received for the last accept event. 


2.2. Message Validity 

Starting from the last transmission of the Resync message consecutive Affirm messages 
are transmitted at Aaa intervals, where Aaa > (D + d). In [Malekpour 2006B, 2006C] A RRmin is 
defined to be = 2 FAaa + 1 clock ticks. At the receiving nodes, the following definitions 

hold: 


- A message ( Resync or Affirm) from a given source is valid if it is the first message from 
that source. A message shall remain valid for the duration of one Aaa- 

- An Affirm message from a given source is early if it arrives earlier than (Aaa - d) after 
previous valid message (Resync or Affirm ) from the same source. 

- A Resync message from a given source is early if it arrives earlier than A RRjmn after 
previous valid Resync message from the same source. 

- An Affirm message from a given source is valid if it is not early. 

- A Resync message from a given source is valid if it is not early. 


2.3. System Assumptions 

1 . The cause of transient faults has dissipated. 

2. All good nodes actively participate in the self-stabilization process and correctly execute 
the protocol. 

3. At most F of the nodes are faulty. 

4. The source of a message is distinctly identifiable by the receivers from other sources of 
messages. 

5. A message sent by a good node will be received and processed by all other good nodes 
within Aaa, where Aaa >(D + d ). 
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6. The initial values of the state and all variables of a node can be set to any arbitrary value 
within their corresponding range (In an implementation, it is expected that some local 
capabilities exist to enforce type consistency of all variables.) 


2.4. Protocol Functions 

The functions used in this protocol are described in this section. 

Two functions InvalidAjfirm( ) and InvalidResync() are used by the monitors. The 
InvalidAffirmO function determines whether or not a received Affirm message is valid. The 
InvalidResync( ) function determines if a received Resync message is valid. When either of these 
functions returns a true value, it is indicative of an unexpected behavior by the corresponding 
source node. 

The Accept() function is used by the state machine of the node in conjunction with the 
threshold value T A = G - 1. When at least T A valid messages ( Resync or Affirm ) have been 
received, this function returns a true value indicating that an accept event has occurred and such 
an event has also taken place in at least F other good nodes. When a node accepts, it consumes 
all valid messages used in the accept process by the corresponding function. Consumption of a 
message is the process by which a monitor is informed that its stored message, if it existed and 
was valid, has been utilized by the state machine. 

The Retry () function determines if at least T R other nodes have transitioned out of the 
Maintain state, where T R = F +1. When at least T R valid Resync messages from as many nodes 
have been received, this function returns a true value indicating that at least one good node has 
transitioned to the Restore state. This function is used to transition from the Maintain state to the 
Restore state. 

The Transitory ConditionsMet() function determines proper timing of the transition from 
the Restore state to the Maintain state. This function keeps track of the accept events, by 
incrementing the Accept_Event_Counter, to determine if at least IF accept events in as many A aa 
intervals have occurred. It returns a true value when the transitory conditions are met. 

The TimeOutRestore() function uses P R as a boundary value and asserts a timeout 
condition when the value of the StateJTimer has reached Pj. Such a timeout triggers the node to 
reengage in another round of self-stabilization process. This function is used when the node is in 
the Restore state. 

The TimeOutMaintain() function uses Pm as a boundary value and asserts a timeout 
condition when the value of the StateJTimer has reached P M - Such a timeout triggers the node to 
reengage in another round of synchronization. This function is used when the node is in the 
Maintain state. 

In addition to the above functions, the state machine utili z es the TimeOutAcceptEvent( ) 
function. This function is used to regulate the transmission time of the next Affirm message. 
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This function maintains a DeltaAA_Timer by incrementing it once per local clock tick and once 
it reaches the transmission time of the next Affirm message, A aa , it returns a true value. In 
response to such a timeout, the node broadcasts an Affirm message. 


2.5. The Self-Stabilizing Clock Synchronization Problem 

To simplify the presentation of this protocol, it is assumed that all time references are 
with respect to a real time to, where to = 0 when the system assumptions are satisfied, and for all 
l > to the system operates within the system assumptions. Let 

• C be the bound on the maximum convergence time, 

• AutcaPTimeM), for real time t, the maximum difference of values of the local timers of any 
two good nodes TV/ and TV/, where TV/, TV/ e K ( „ and K ( , is the set of all good nodes, and 

• A p rec i S i on , also referred to as self-stabilization precision, the guaranteed upper bound on the 
maximum separation between the local timers of any two good nodes TV, and TV, in the 
presence of a maximum of F faulty nodes, where TV/, TV, e K ( ,. 

A good node TV, resets its variable Local_Timen periodically but at different points in 
time than other good nodes. The difference of local timers of all good nodes at time t, 
ALocai_Timer(t), is determined by the following equation while recognizing the variations in the 
values of the Local_Timeri across all good nodes. 

A Lo cai_Timer( t) = min ( ( Local ffT ’imer max ( t) - Local _T imer min ( t)), 

( Local Timer max(t - /~Ap r ecision J) Local Timer min(t - f Aprecision 1})), 

where, 

Local_Timer min (x) = min ({Local_Timerj(x) I TV/ e K G J), 

Local _Timer max (x) = max ({Local_Timeri(x) I TV, e Kg}), and 


There exist C and Ap recision : 

Convergence: Ap oca i_pi mer (C) — Ap rec i S i on 
Closure: A i , t AC, Ap oca i_pj mer (t) A Ap rec j S { on 

The values of C, Ap re cision, and the maximum value for Local_Timerj, Local_Timer_Max, are 
determined to be: 

C = (2 P t+ Pm) A^, 

Aprecision — (3 F - 1) A A/ \ D + Ap> r ift, 

Local_Timer_Max = Pp + Pm, 

and the amount of drift from the initial precision is given by 

Aorift = ((1 +p) - 1/(1 +p)) P Effective Aaa. 


Note that since Local_Timer_Max > P/ /2 and since the Local_Timer is reset after reaching 
Local_Timer_Max (worst case wraparound), a trivial solution is not possible. 
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2.6. The Byzantine-Fault-Tolerant Self-Stabilizing Protocol for Distributed Clock 
Synchronization Systems 


The presented protocol is described in Figure 3 and consists of a state machine and a set 
of monitors which execute once every local oscillator tick. 


Monitor: 

Affirm: 

case (incoming message from the 

if InvalidAjfirm( ) then 

corresponding node) 

Invalidate the message 

{Resync: 

else 

if InvaliclResyncO then 

Validate and store the message. 

Invalidate the message 

Other: 

else 

Do nothing. 

V alidate and store the message, 
Set state status of the source. 

} // case 


Node: 

Maintain: 

case (state of the node) 

if TimeOutMaintain() or Retry() then 

{Restore: 

Transmit Resync message, 

if TimeOutRestore() then 

Reset State _Timer, 

Transmit Resync message, 

Reset DeltaAA_Timer, 

Reset State _Timer, 

Reset Accept _Event_Counter, 

Reset DeltaAA_Timer, 

Go to Restore state, 

Reset Accept_Event_Counter, 


Stay in Restore state, 

elsif TimeOutAcceptEventO then 
if Accept( ) then 

elsif TimeOutAcceptEvent( ) then 

Consume valid messages., 

Transmit Affirm message, 

if ( State _Timer = [ A Precisio J) 

Reset DeItaAA_Timer, 

Reset Local _Timer., 

if Accept( ) then 

Transmit Affirm message, 

Consume valid messages, 

Reset DeltaAA_Timer, 

Clear state status of the sources, 

Stay in Maintain state, 

Increment Accept _Event_Counter, 


if TransitoryConditionsMet() then 

else 

Reset State _Timer, 

Stay in Maintain state. 

Go to Maintain state, 

} // case 

else 

Stay in Restore state. 

else 

Stay in Restore state., 

else 

Stay in Restore state. 


Figure 3. The self-stabilization protocol. 
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2.7. Semantics of the pseudo-code 

• Indentation is used to show a block of sequential statements. 

• Y is used to separate sequential statements. 

• ‘.’is used to end a statement. 

• is used to mark the end of a statement and at the same time to separate it from other 
sequential statements. 

3. Mechanical Verification 

Several approaches were explored toward the mechanical verification of the correctness 
of the initial design of this protocol. This effort started, chronologically, by simulation of the 
known cases and grew into model checking of all scenarios using various model-checking tools. 
Initially, verification of a self-stabilizing protocol for a 4-node system seemed deceptively trivial, 
but in time its complexity became clearer. 

The initial model of the 4-node system required more memory for the construction of the 
state space than the available 2GB of memory. As a result, many abstractions were made and a 
number of reduction techniques were devised to circumvent the state space explosion problem. 
Some of the techniques used are explained in the following sections. 


3.1. Simulation 

The first mechanical verification was accomplished using a VHSIC Hardware 
Description Language (VHDL) 1 implementation that verified the proper operations of the 
protocol for specific cases. The VHDL tools run on a PC with 1GB of memory under the 
Windows 2000 operating system. The VHDL environment is primarily suited for simulation of 
specific scenarios where examination of the known cases requires proper set up of the system for 
each case, separately. The simulation effort provided the sanity checks needed to embark into 
more complex model checking efforts. Nevertheless, within the simulation environment, proper 
operation of the protocol under fault-free conditions were examined and verified. Proper 
operation of the protocol in the presence of faults and for the known scenarios were also 
examined and verified. As the number of cases to be examined increased, this process became 
impractical. As a result, and in an effort to examine all possible scenarios, this approach was 
abandoned in favor of symbolic model checkers. 


3.2. SMV 

The Symbolic Model Verifier (SMV) was used in the second attempt at modeling of this 
protocol on a PC with 2GB of memory running Linux. SMV allows the designers to formally 
verify temporal logic properties of finite state systems. Developers use SMV to verify the design 
for all possible input sequences, instead of a chosen selection of sequences as in simulation. 


1 Very High Speed Integrated Circuit (VHSIC) Hardware Description Language. 
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SMV’s language description and modeling capability provide relatively easy translation from 
VHDL. SMV also provides the desired capability to introduce randomness into the initial values 
of the variables. Despite many abstractions employed, the model’s large state space was beyond 
SMV’s capability for the available platform. In fact, the amount of memory needed for the 
construction of the Binary Decision Diagram (BDD), approximately 10 44 initial states, readily 
exceeded the 2GB available on the PC after a few steps. To further reduce the state space, only a 
subset of critical scenarios was selected. Although this subset was much larger than the number 
of simulation cases, it still lacked the full coverage needed to rule out unforeseen scenarios. 

Clearly, more memory and computing power were needed. A new PC with 4GB of 
memory running Linux was purchased. Once again, the amount of memory needed by SMV 
readily exceeded the 4GB available memory. 


3.3. SMART 

The next modeling effort of this protocol was in Stochastic Model checking Analyzer for 
Reliability and Timing (SMART) [Ciardo 2003] on a PC with 4GB of memory running Linux. 
SMART is a software package that integrates various high-level logical and stochastic modeling 
formalisms (e.g., stochastic Petri nets) in a single modeling study. For model checking, SMART 
uses Multi Decision Diagram (MDD) to store large sets of states, a Kronecker matrix encoding 
of transition relation between state, and the saturation algorithm for state space construction 
[Siminiceanu 2004]. This symbolic approach can manage the memory consumption problem in 
a more efficient manner. Unlike SMV, SMART lacks an intuitive interface, thus, using it 
requires greater level of expertise. Unfortunately, due to the complexity of the protocol, the 
analysis of the model in SMART also exceeded the 4GB available memory and could not fully 
examine all possible cases in a reasonable amount of time. Nevertheless, using SMART, more 
scenarios were examined than with SMV and the protocol was demonstrated to be self- 
stabilizing as expected. Many attempts were made to get around the limitations, but at the end 
this effort was also abandoned. 


3.4. SMV Revisited 

The intuitive solution to this problem is to provide more memory. There is a hardware 
limitation on the amount of memory that can be added to a given system. Furthermore, although 
additional memory would ease the state space construction, it may not eliminate the problem. 

Another solution, if there is one, is to redesign the protocol. What is presented in 
[Malekpour 2006B and 2006C] and model checked here is the redesigned version of the 
protocol. The amount of memory needed to fully model check the general case of this protocol 
far exceeds the available 4GB of memory. Nevertheless, the protocol can now be exhaustively 
model checked for a 4-node system. 
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4. Modeling Simplifications and Abstractions 

The local measures within each node are used to keep track of timing of the self- 
stabilization events. Although the derived parameters are defined with respect to the real time, 
ultimately, in implementations they have to be translated into discrete values. Discretization of 
the derived parameters is performed using the ceiling operation. In this protocol, all local 
variables and watchdog timers are discretized and represented by integer values. These local 
variables are, therefore, measured with respect to the local clock. 

The state space for modeling of the general case of this protocol far exceeds the available 
4GB memory. Thus, in a bottom-up approach, a basic case is modeled such that the number of 
parameters needed are minimal and the range of each parameter is at its minimum. A distributed 
system tolerating as many as F Byzantine faults requires a network size of more than 3 F nodes 
[Lamport 1982, Lamport 1985] to maintain synchrony. In other words, to guarantee the closure 
property a minimum of 3F+ 1 nodes are needed. Therefore, the basic case is defined as the 
minimum number of nodes that can self-stabilize in the presence of at least one Byzantine faulty 
node and with all other parameters at their minimum. Thus, for the basic case , the number of 
nodes in the system K = 4, the upper bound on the number of faulty nodes F = 1, and the 
minimum number of good nodes, G, is determined to be G = K - F = 3 nodes. 

Other aspects of the basic case are topological issues. The logical topology is a fully 
connected graph of a 4-node system, where each node is directly connected to another node via a 
dedicated bi-directional channel. As shown in Figure 4, each node and the source of a message 
is distinctly identifiable by other nodes. The physical topology can be either a fully connected 
graph, similar to the logical topology, or equivalently, a graph where a message from a source is 
broadcast to all other nodes at the same time. For the basic case, broadcast is modeled using a 
single variable. 



Figure 4. A 4-node system. 

Recall that all parameters are defined as integers. The event response delay, D, and the 
network imprecision, cl, are chosen to be at their minimum values of 1 and 0 clock ticks, 
respectively. As a result, Aaa is at its minimum of one clock tick. This simplification, 
consequently, implies that the logical timers of the good nodes are in phase with each other. 
Note that this simplification does not imply that the nodes are synchronized with each other. To 
further minimize the state space, the clock drift rate, p, is chosen to be zero. This simplification 
guarantees that the nodes’ State_Timer will remain in phase with each other. Model checking of 
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the system with Aaa > 1 where the logical timers of the good nodes are in phase with respect to 
each other, is equivalent to model checking for Aaa = 1 and the basic case. However, model 
checking of the system with Aaa > 1, where the logical timers of the good nodes are out-of-phase 
with respect to each other, poses a greater challenge. 

We recognize that the choice of the value for network imprecision, d = 0, is a nonrealistic 
assumption. Nonetheless, these simplifications are necessary in order to reduce the state space to 
a manageable size. Furthermore, we believe that the basic case specifies the set of necessary 
conditions that all candidate solutions to this problem should satisfy. As an example, the flaw in 
[Daliot 2003] was discovered as a direct result of applying that protocol to the basic case as 
documented in [Malekpour 2006A]. We also acknowledge that satisfying the basic case does 
not necessarily imply that the candidate solution solves the general case of this problem. 

In order to expedite the self-stabilization process, in general, and in order to minimize the 
state space for model checking purposes, in particular, the convergence time has to be 
minimized. It was argued in [Malekpour 2006B] that Pzmin = 10 and P M ^ Pt- Although the 
maximum duration of the Restore state, Pt, can be any value larger than the required minimum, 
P T is chosen to be P/.min- In order to minimize the state space, P M is chosen to be equal to P T . 
Therefore, synchronization period, P, for the basic case is chosen to be P = Pm = Pt = 10. For 
the basic case, the parameters d and p are chosen to be zeros. In other words, there are no 
variations in the communication delay and the nodes do not drift with respect to each other. 
Model checking of the system with larger values for Pm and Pt is equivalent to model checking 
for P = P M = Pt = 10. 

A system clock, SCLK, is introduced to keep track of passage of time from the global 
perspective. The SCLK is managed at the system level and is incremented per SMV cycle. Each 
node has a logical clock, Local_Timer, that locally keeps track of time. This logical clock is 
used to measure the convergence time, C, as well as the self-stabilization precision, A Prec i sion , 
across good nodes (i.e. external view of the system). Since for the basic case the logical timers 
(. StateJTimer and Local _Timer) of the good nodes are in phase with each other and since Aaa = 1 
and p = 0, a single SCLK suffices to drive timers of all nodes. The use of a single SCLK also 
eliminates redundancies at the node level for replicating behavior of local oscillators and, thus, 
reduces the state space substantially. The SCLK, therefore, binds the whole system together, 
providing a means for advancing the StateJTimer and LocalJTimer at the node and an external 
view of the system at any time. Although the use of a single clock does not imply synchrony at 
the nodes, it does imply that the nodes are in phase with each other at the StateJTimer and 
LocalJTimer levels. However, due to the inherent randomness of the operation of the model 
checkers, the order of execution of the nodes is not predetermined. Since there is no control over 
the order of transmission of messages and the start of execution of the nodes at each model 
checker cycle, the nodes potentially broadcast and receive messages out of order of issuance. 
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5. Modeling the System 


To accommodate for proper timing of operations of the system, variables are needed to 
keep track of passage of time in each monitor and node. Introduction of such variables 
exponentially increases the state space beyond the 4GB available memory. For the general case 
of modeling this protocol, a Transmit _Timer is needed at every node to regulate proper timing of 
outgoing messages. A Receive_Timer is needed at each monitor to keep track of proper timing 
of incoming messages from its corresponding source [Malekpour 2006B]. As Aaa increases 
linearly, the state space associated with Transmit _Timer and Receive_Timer increases 
exponentially. 

There are two different ways of modeling this protocol, either all operations are done 
sequentially in one big module, or the operations are partitioned between the node and its 
monitors. In a sequential model, all activities take place within the same scope and during one 
clock tick. Such a model is not readily scalable. A modular model is readily scalable, but 
requires coordinated interactions between the node and its monitors. Either the monitors have to 
inform the node of the changes in their current status or the node has to poll the status of the 
monitors to stay current with the changes in the system. In turn, the monitors have to be 
informed by the node to take certain actions at the appropriate time. Since the node and its 
monitors operate with respect to a local clock, there will be a delay in a monitor’s response to the 
node’s commands. The interactions between the node and its monitors can be coordinated either 
based on time or by passing a control token in a master-target fashion. 

In this SMV model, a modular approach is employed where the interactions between a 
node and its monitors are coordinated based on time. Also, to minimize the state space both 
positive and negative edges of the SCLK are used. In particular, the nodes operate at the positive 
edge of the SCLK while the monitors operate at the negative edge of the SCLK. For A aa = 1, 
operating at the positive edge of the SCLK, the nodes are guaranteed not to violate the minimum 
transmission time requirement for their consecutive output messages. Therefore, for the basic 
case there is no need for the Transmit _Timer variable and, consequently, no need for the 
Receive_Timer variable. Thus, further reduction in memory and computation requirements is 
achieved. Since A aa = D = 1 and A Dri f t = 0, 

4 Precision - (3 F 1) Aaa - D + Aj ) r ; A — —A \ \ - D + 0 — -1 \ \ - and 

r Ap rec i S i on \ — Aaa — 1 • 

Since Aaa = 1 and P T = P M = P = 10, 

C = (2Pt + P M ) Aaa = 3P = 304u = 30. 
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6. Models and Data Structures 


In this section, the system components are modeled and subsequently their data structures 
are defined. 


6.1. Modeling Faulty Nodes 

The fault tolerant requirement of K > 3F+1 implies that the system of 4 nodes can tolerate 
up to one Byzantine faulty node. Therefore, the system is devised to consist of 3 good nodes and 
one faulty node. In Figure 5 the faulty node, N 4 , is shown in gray. 



Figure 5. A 4-node system with a faulty node. 

To properly portray the behavior of the faulty node, Figure 5 needs to be redrawn. Figure 
6 portrays a symmetric faulty node and a crash-silent node that is a special case of a symmetric 
faulty node where every good node, Nj through N3, have the same view of the faulty node, N 4 . 



Symmetric faulty 


Crash- silent 


Figure 6. A 4-node system with a symmetric faulty node. 


Modeling of an asymmetric (Byzantine) faulty node is more complex than the symmetric 
faulty node. The malicious nature of the Byzantine faulty node is such that as if each good node 
is affected independently by the Byzantine faulty node. Such behavior of the Byzantine faulty 
node is depicted in Figure 7 by replicating the effects of the Byzantine faulty node, N4, for each 
good node Nj through N3. Furthermore, the Byzantine faulty behavior modeled here is a node 
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with arbitrarily malicious behavior. Defined earlier as permanent Byzantine faulty, the 
Byzantine faulty node is allowed to influence other nodes at every clock tick and at all time. 



Figure 7. A 4-node system with an asymmetric (Byzantine) faulty node. 

Since the behavior of a faulty node is not the same as a good node, modeling of a faulty 
node requires rethinking. Proper modeling of faulty nodes can potentially result in considerable 
state space reduction. It particular, a Byzantine faulty node may transmit any one of the three 
possible messages, namely, NONE, Resync, or Affirm at any time. Additionally, unlike the good 
nodes, local state of a faulty node does not play a role in the operation of this protocol. 
Therefore, the faulty node is modeled as a special node only capable of randomly producing any 
one of the three messages at any clock tick and without any internal state. Consequently, the 
faulty node’s data structure has only one parameter. Message _Out. The range of values that this 
element can hold is enumerated as follows. 

Message_Out = {NONE, Resync, Affirm} 


6.2. Modeling Monitors 

The assessment results of the monitored nodes are utilized by the node in the self- 
stabilization process. The node consists of a state machine and a set of {K -1) monitors. The 
state machine describes the collective behavior of the node, A„ utilizing assessment results from 
its monitors, Mj .. M^i, M i+/ .. M K as shown in Figure 8, where Mj is the monitor for the 
corresponding node Nj. 
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Node 



Figure 8. Interaction of the node’s state machine and its monitors. 

A monitor keeps track of activities of its corresponding source node. A monitor detects 
proper sequence and timeliness of the received messages from its corresponding source node. A 
monitor reads, evaluates, time stamps, validates, and stores only the last message it received 
from that node. A monitor also keeps track of the state of the source node by keeping track of 
received Resync messages, separately. The monitor’s data structure consists of Last_Message, 
Receive_Timer, MessageJValid, Delta_RR_Timer, and Received _Resync . The Last_Message 
element represents the last valid message received from the corresponding source node. The 
Receive_Timer element represents the time interval between arrival of the last two messages 
from the corresponding source node. As discussed in the previous section, there is no need to 
model this element for the basic case. The Message Valid element indicates whether or not the 
last message received was valid. The Delta_RR_Timer element represents the duration of time 
between any two consecutive valid Resync messages from the corresponding source. The 
Received_Resync element indicates whether the last valid message received was a Resync 
message. The range of values that these elements can hold is enumerated as follows. 


Last_Message 
Receive_Timer 
Message _Valid 
Delta_RR_Timer 
Received_Resync 


- {Resync, Affirm} 

= {0..(4u+l)} 

= { 0 , 1 } 

= {0..(/V + P M )} 

= { 0 , 1 } 


6.3. Modeling Good Nodes 

The state machine describes the collective behavior of the node, Nj, utilizing assessment 
results from its monitors, Mj .. Mm, M;+i .. Mk as shown in Figure 8. The good node’s data 
structure consists of State, Accept _Events, State_Timer, Local_Timer, Transmit _Timer, and 
Message_Out. The State element represents the current state of the node. The Accept_Events 
element is the count of accept events since the node entered the Restore state. The State_Timer 
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element represents the duration of current state of the node. The Local_Timer element represents 
the duration of time since the node has been synchronized with other good nodes. The 
Transmit _Timer element represents the passage of time since the transmission of the last 
message by the node. As discussed in the previous section, there is no need to model this 
element for the basic case. The Message_Out element represents the out going message of the 
node. The range of values that these elements can hold is enumerated as follows. 


State 

Accept_Events 
State _Timer 
Local_Timer 
Transmit _Timer 
Message_Out 


= { Restore , Maintain} 

= {0..(F+1)} 

= {0 .. P M ) 

= { 0 .. (Pt + Pm)} 

= {0..(4u+l)} 

= { NONE, Resync, Affirm } 


6.4. Modeling Communication Channels 

The communication channel’s data structure consists of Message_In, Comm_Delay, and 
Message_Out. The Message_bi element represents the message deposited by the transmitting 
node. The Comm_Delay represents the amount of delay associated with the channel. The 
Message_Out element represents the delayed message being delivered to the destination nodes. 
The range of values that these elements can hold is enumerated as follows. 

Message _fn = {NONE, Resync, Affirm} 

Comm_Delay = {1 .. A aa } 

Message_Out = {NONE, Resync, Affirm} 

Since for the basic case Aaa is one clock tick, a deposited message on a communication 
channel is available to the destination nodes at the next clock tick. Therefore, a channel of depth 
one suffices. Also since a message is broadcast to other nodes, a single variable suffices to 
represent the communication channel from a node to all other nodes. Therefore, in order to 
reduce the state space, the communication channel is modeled implicitly and as part of the 
node’s out going message instead of introducing a new SMV module for the channels. 
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7. Propositions 

Computational tree logic (CTL), a temporal logic, is used to express properties of a 
system in this context. CTL uses atomic propositions as its building blocks to make statements 
about the states of a system. CTL then combines these propositions into formulas using logical 
and temporal operators with quantification over runs. The CTL operators have the following 
format. 

Q T 

there exists an execution E X next 

for all executions A F finally (eventually) 

G globally 

U until 

In this section the claims of convergence and closure properties as well as the claims of 
maximum convergence time and determinism of the protocol for the basic case are examined. 
Although in the description of the protocol these properties are stated separately, nevertheless, 
they are examined via one CTL proposition. Validation of this general CTL proposition requires 
examination of a number of underlying propositions. In particular, since Au, C ai_Timer(t) is defined 
in terms of the LocalJTimer of the good nodes and the Local_Timer is defined in terms of the 
State_Timer, examination of the properties that described proper behavior of the State JTimer 
take precedence. As a result, in this section, the four underlying propositions are examined 
followed by the general proposition that validates the convergence and closure properties of the 
protocol as well as the claims of maximum convergence time and determinism. 

The following properties are described with respect to only one good node, namely 
Good_Node_l. Since all good nodes are identical, due to the symmetry, the result of the 
propositions equally similarly applies to other good nodes. 

Proposition 1: This property specifies whether or not the State, JTimer of a good node takes on a 
given value in its range infinitely often, for instance, its maximum value of P. The expected 
result for this proposition is a true value. 


AF (Good_Node_l.State_Timer = P) 


Examining the negation of this property is expected to produce a false value. This proposition 
verifies that the State_Timer of a good node cannot never reach a given value. 


EG !(Good_Node_l.State_Timer = P) 


Similar properties apply to the LocalJTimer, but within its expected range. 
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Proposition 2: This property specifies whether or not the State _Timer of a good node takes on 
all values in its range infinitely often. In other words, it verifies that the model does not 
deadlock. Furthermore, the value of the State Jimer of a good node at the next clock tick is 
different from its current value and is its expected next value in the sequence of 0 to P. The 
expected result for this proposition is a true value. 


AG (((SCLK = 1) & (Good_Node_l.State_Timer = i)) -> 

AX ((SCLK=0) & ((Good_Node_l.State_Timer= i) I (Good_Node_l.State_Timer = i+1)))) & 
AG (((SCLK = 1) & (Good_Node_l.State_Timer = P)) -> 

AX ((SCLK = 0) & (Good_Node_l.State_Timer = 0))) 

For all i = 0 .. (P-1) 


Examining the negation of this property is expected to produce a false value. This proposition 
verifies that the next value of the State JTimer of a good node cannot be the same as its current 
value. In other words, its value always advances within the expected range. 


EG (((SCLK = 1) & (Good_Node_l.State_Timer = i)) -> 
EX ((SCLK = 0) & (Good_Node_l.State_Timer = i))) I 

For all i = 0 .. (P-1) 


Similar properties apply to the Local_Timer, but within its expected range. 


Proposition 3: This property specifies whether or not time advances and the amount of time 
elapsed, ElapsedJTime, has advanced beyond the predicted convergence time, 
Convergence_Time. The expected result for this proposition is a true value. 


Elapsed_Time := (Global_Clock >= Convergence_Time) ; 
AF (Elapsed_Time) 


The Global_Clock is a measure of elapsed time from the beginning of the operation and with 
respect to the real time, i.e. external view. The Elapsed_Time is indicative of the Global_Clock 
reaching its target maximum value of Convergence_Time. 

init (Global_Clock) := 0 ; 
next (Global_Clock) := 
case 

(SCLK = 1) & (Global_Clock < ConvergenceJTime) : Global_Clock + 1 ; 

1 : Global_Clock ; 
esac ; 

ElapsedJTime := (Global_Clock >= ConvergenceJTime) ; 


18 









Proposition 4: Similar to Proposition 2, this property specifies whether or not the State_Timer 
of a good node takes on all values in its range infinitely often but beyond the convergence time, 
i.e. after Elapsed_Time has become true. The expected result for this proposition is a true value. 
Examining the negation of this property is expected to produce a false value. Similar properties 
apply to the LocaI_Timer, but within its expected range. 


AF (Elapsed_Time) & 

AG (((SCLK = 1) & (Elapsed_Time) & (Good_Node_l.State_Timer = i)) -> 

AX ((SCLK=0) & ((Good_Node_l.State_Timer= i) I (Good_Node_l.State_Timer = i+1)))) & 
AG (((SCLK = 1) & (Elapsed_Time) & (Good_Node_l.State_Timer = j)) -> 

AX ((SCLK = 0) & (Good_Node_l.State_Timer = j+1))) & 

AG (((SCLK = 1) & (Elapsed_Time) & (Good_Node_l.State_Timer = P)) -> 

AX ((SCLK = 0) & (Good_Node_l.State_Timer = 0))) 

For all i = 0 .. 4 
For all j = 5 .. (P-1) 


Proposition 5: The convergence and closure properties are described in Section 2.5. This 
proposition encompasses the criteria for the convergence and the closure properties as well as the 
claims of maximum convergence time and determinism. This proposition specifies whether or 
not the system will converge to the predicted precision after the elapse of convergence time, 
ElapsedJTime, and whether or not it will remain within that precision thereafter. The expected 
result for this property is a true value. 


AF (ElapsedJTime) & 

— Determinism Property 

AG (ElapsedJTime -> All_Within_Precision) & 

— Convergence Property 

AG ((ElapsedJTime & All_Within_Precision) -> 
AX (ElapsedJTime & All_Within_Precision)) 

— Closure Property 


The proper value of the All_Within_Precision is determined by measuring the difference of 
maximum and minimum values of the Local_Timers of all good nodes for the current SCLK tick 
and in conjunction with the result from the previous SCLK tick. The expected difference of 
LocalJTimers is the predicted precision bound. 

The negation of the above proposition is listed below and the expected result is a false value. 
This property specifies that after the elapse of convergence time, ElapsedJTime, whether or not 
the system will not converge or if it converges, whether or not it drifts apart beyond the expected 
precision bound. 


AF (Elapsed_Time) & 

AG (ElapsedJTime -> All_Within_Precision) & 

AG ((ElapsedJTime & All_Within_Precision) -> EX (! All_Within_Precision)) 
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8. Results 


This SMV model checking effort was performed on a PC with 4GB of memory running 
Linux. SMV was able to examine all possible scenarios and the basic case of the protocol was 
model checked. The model checking results are listed in the following tables. The negation of a 
property is denoted by using the unary operator T. 

The Byzantine faulty behavior modeled here is a node with arbitrarily malicious 
behavior. The Byzantine faulty node is allowed to influence other nodes at every clock tick and 
at all time as depicted in Figure 7. Regardless of the nature of the faulty node, no assumptions 
are made about the initial internal status of the nodes, the monitors, and the system. For 
instance, a node can wake up in the Maintain state and transmit a Resync, message. Although 
such behavior from a good node is not exhibited during normal operation, nevertheless, it is 
allowed for the random start up. Such a model is for the weakest assumptions about the behavior 
of the faulty nodes, the internal state of data structures of the nodes, the monitors, and the system 
as a whole, and thus produces the strongest results. 

Table 1. Results in the presence of a Byzantine faulty node. 


Proposition 

Result 

Time (sec) 

Mem (GB) 

1 

T 

1311 

1.2 

1! 

F 

1318 

1.2 

2 

T 

0.2 

0.012 

2! 

F 

8866 

1.2 

3 

T 

0.04 

- 

4 

T 

19 

0.056 

4! 

F 

4702 

1.2 

5 

T 

2313 

2 

5! 

F 

3413 

2.1 


Table 1 lists the results of model checking of the basic case for the stated propositions 1 
through 5, where the duration of the Maintain and Restore states, Pm and Pt, are chosen to be 
Pm = Pt = Period =10 and the maximum convergence time, Convergence_Time, is 30. As 
shown in Table 1, the maximum memory usage is about 2GB after applying the state space 
reduction techniques. The amount of memory used and processing time needed depend on the 
BDD construction and the nature of the query. Although verification of the stated propositions 
suffices to validate the claims of correctness and determinism of the protocol and in the presence 
of a Byzantine fault, the propositions are further examined for other, and hence less severe, types 
of faults. For the following scenarios, the values for the Period and Convergence_Time are the 
same as for Table 1. 
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8.1. Symmetric Fault 


In this case, all good nodes receive identical messages from a single faulty node as 
depicted in Figure 6. The faulty node still behaves randomly, but its effect at the receiving nodes 
is identical. As shown in Table 2, the maximum available memory is used to model check this 
case. Due to the BDD construction, the memory usage is far more than the Byzantine faulty case. 

Table 2. Results in the presence of a symmetric faulty node. 


Proposition 

Result 

Time (sec) 

Mem (GB) 

1 

T 

2573 

2.0 

2 

T 

0.2 

0.012 

3 

T 

0.04 

- 

4 

T 

62 

0.160 

5 

T 

3975 

3.5 


Of 4GB available memory, maximum memory utilized by SMV is approximately 3.5GB. 

8.2. Crash-Silent Fault, a.k.a. Stuck-at NONE Message 

This case is a special case of the symmetric faulty node where the faulty node is not 
transmitting any messages. This case is modeled such that the associated message from the 
faulty node to all good nodes is a NONE message signifying lack of transmission by the faulty 
node. This case is depicted in Figure 6. 

Table 3. Results in the presence of a symmetric faulty node. 


Proposition 

Result 

Time (sec) 

Mem (GB) 

1 

T 

28 

0.045 

2 

T 

0.15 

- 

3 

T 

0.04 

- 

4 

T 

6 

0.015 

5 

T 

365 

0.34 


8.3. Stuck-at Resync Message 

This case is another special case of the symmetric faulty node where all good nodes 
receive identical messages from a single faulty node. The faulty node transmits the same 
message to all good nodes all the same time. 
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Table 4. Results in the presence of a symmetric faulty node. 


Proposition 

Result 

Time (sec) 

Mem (GB) 

1 

T 

81 

0.25 

2 

T 

0.15 

- 

3 

T 

0.04 

- 

4 

T 

7 

0.025 

5 

T 

605 

0.61 


8.4. Stuck-at Affirm Message 

This case is another special case of the symmetric faulty node where all good nodes 
receive identical messages from a single faulty node. The faulty node transmits the same 
message to all good nodes all the same time. 

Table 5. Results in the presence of a symmetric faulty node. 


Proposition 

Result 

Time (sec) 

Mem (GB) 

1 

T 

19 

0.033 

2 

T 

0.15 

- 

3 

T 

0.04 

- 

4 

T 

5 

0.017 

5 

T 

276 

0.3 


9. Additional Reduction Techniques 

New state space reduction techniques are presented here that can be used in mechanical 
verification of other protocols. Although these techniques were not used in the model checking 
efforts reported here, they are intended to be used in the future efforts. The underlying 
assumption for these state space reduction techniques is that a message from a good node will 
eventually (see requirements for message validity for this protocol) be accepted as valid. Since 
this assumption is true for the good nodes and once true they do not violate the message timing 
requirements, the associated monitors for the corresponding good nodes can be simplified so that 
they do not have to examine proper timing of message arrival. 

In the SMV model reported here, the faulty node is modeled as a special node only 
capable of randomly producing any one of the three messages at any time. Per protocol 
requirements, a good node must keep track of the incoming messages from all other nodes. 
Therefore, K - 1 monitors at each good node are needed to accommodate this requirement. 
Hereafter, such straightforward model of a faulty node is referred to as explicit fault model and 
the associated monitors as explicit fault monitors. 
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Recall that the Accept() function uses the threshold value Ta = G - 1 = IF where 
potentially up to F of these messages are from as many faulty nodes. Looking from a different 
perspective, at least F of these messages have to be from as many good nodes. Similarly, the 
Retry() function uses the threshold value T R = F + 1 and potentially up to F of these messages are 
from as many faulty nodes. In other words, at least one of these messages have to be from a 
good node. Now, let’s assume that a good node receives messages only from the other good 
nodes. In this case, for the Accept() function, unless the node receives at least F messages, no 
matter how many messages (up to F) from the faulty nodes are assumed to be present, the 
Accept( ) function will not return a true value. Similarly, for the Retry() function, unless the node 
receives at least one message, no matter how many messages (up to F) from the faulty nodes are 
assumed to be present, the Retry( ) function will not return a true value. 

After receiving at least F messages from as many good nodes for the Accept( ) function 
and at least one message from a good node for the Retry( ) function, the behavior of the faulty 
nodes can either strengthen a good node’s current status or cause the good node to lose 
synchronization with other nodes. Therefore, only at such moments does the behavior of the 
faulty nodes impact the operations of the good nodes and, thus, the behavior of the faulty nodes 
can be inferred as needed at the good nodes. Exploiting this concept reveals that the faulty 
nodes, the associated explicit fault monitors for the corresponding faulty nodes, and the 
corresponding communication channels are no longer needed. Hereafter, such an indirect model 
of a faulty node is referred to as an implicit fault model. This concept is depicted in Figure 9 
where the good nodes are denoted by N/ .. Nm, N i+] .. N K -f and their associated explicit monitors 
are denoted by Mi .. Mm, M i+] .. M K -f and the monitors M K -f+i ■■ M K represent the implicit fault 
models. 


Explicit 


Implicit 


Node j 



Figure 9. Implicit fault model. 
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In the implicit fault model approach a good node receives messages only from other good 
nodes and after accumulating enough messages (F for the T A and one for T R ), the good node’s 
subsequent behavior will be determined by randomly introducing up to F messages for the faulty 
nodes. Therefore, in this approach, behaviors of faulty nodes are imitated at the good node and 
when appropriate. Thus, the implicit fault model substantially improves the model checking 
performance. In particular, if a node’s behavior will not be influenced by the behavior of the 
faulty nodes for a duration of time, the model checking time can advance to the end of that time 
interval. This performance increase is more noticeable in protocols that do not require periodic 
transmissions of messages. Also, by eliminating the explicit fault monitors and the associated 
channels, the implicit fault model results in substantial reduction in the state space. 

The implicit fault model can be used directly in protocols that do not require keeping 
track of a history of a node’s behavior. Otherwise, an additional measure is required to 
compensate for the removal of the explicit fault monitors. In particular, for the protocol 
presented in this report, elimination of an explicit fault monitor can be compensated by the 
introduction of a new implicit fault monitor at the node. Such a monitor has to guarantee 
proper timing of any two consecutive actions associated with their corresponding messages. 

Alternatively, the faulty node can be modeled as a special node that is still capable of 
randomly producing any one of the three messages but its outgoing messages are regulated such 
that the message validity requirements of the protocol are not violated. Such a well-behaved 
model of a faulty node is referred to as a semi-explicit fault model. In this approach, the nodes 
are modeled explicitly with K-\ explicit monitors but they assume that all incoming messages 
meet their protocol requirements and, therefore, are valid. Therefore, the model of the monitors 
can be simplified. 

The explicit fault model is simpler to model, easier to scale to a larger system, but 
requires more memory than the implicit fault model. Modeling of the implicit fault model 
requires more care, but the improved performance and the reduction gained in the state space far 
outweigh its added complexity. Because of its simplicity and direct approach and avoiding any 
assumptions regarding message validity, the explicit fault model was used in this verification 
effort. The semi- explicit fault model and implicit fault model will be used in future work. 


10. Applications 

The proposed self-stabilizing protocol is expected to have many practical applications as 
well as many theoretical implications. Embedded systems, distributed process control, 
synchronization, inherent fault tolerance which also includes Byzantine agreement, computer 
networks, the Internet, Internet applications, security, safety, automotive, aircraft, wired and 
wireless telecommunications, graph theoretic problems, leader election, time division multiple 
access (TDMA), and the SPIDER 2 project [Torres 2005A, 2005B] at NASA-LaRC are a few 


2 Scalable Processor-Independent Design for Enhanced Reliability (SPIDER). 
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examples. These are some of the many areas of distributed systems that can use self-stabilization 
in order to design more robust distributed systems. 


11. Summary and Future Work 

In this report a SMV model of a simplified model of a rapid Byzantine-fault-tolerant self- 
stabilizing protocol for distributed clock synchronization systems is presented. The simplified 
model of the protocol is model checked using SMV where the entire state space is examined and 
proven to self-stabilize in the presence of one permanent Byzantine faulty node. Furthermore, 
the simplified model of the protocol is proven to deterministically converge with a linear 
convergence time with respect to the self-stabilization period as predicted. This protocol does 
not rely on any assumptions about the initial state of the system and no assumptions are made 
about the internal status of the nodes, the monitors, and the system as a whole, thus making the 
weakest assumptions and, therefore, producing the strongest results. The Byzantine faulty 
behavior modeled here is a node with arbitrarily malicious behavior. The Byzantine faulty node 
is allowed to influence other nodes at every clock tick and at all time. The only constraint is that 
the interactions are restricted to defined interfaces. 

In this report, modeling challenges are addressed and abstraction techniques are 
illustrated. A number of innovative state space reduction techniques, in particular the implicit 
fault model of the faulty nodes and their corresponding monitors, are introduced that can be used 
in a verification process of other protocols. In addition, the basic case is introduced that 
specifies the set of necessary conditions that all candidate solutions to this problem should 
satisfy. The flaw in [Daliot 2003] was discovered as a direct result of applying that protocol to 
the basic case [Malekpour 2006A], Although model checking results of the basic case of the 
protocol are promising, these results are not sufficient to confirm that the protocol solves the 
general case of this problem. 

Having mechanically verified a simplified model of the protocol, new hypothesis and 
conjectures are now practical for examination. The current modeling approach is a very 
powerful tool for asking “What if?" questions that are difficult to answer either by manual 
analysis or by testing real hardware. 

In our ongoing efforts toward the verification of this protocol for the general case, the 
SMV model of the simplified version of this protocol has been redesigned and restructured. 
Also, the protocol has been redesigned and further simplified. As a result, the current model 
requires less memory, making exploration of more complex and larger configurations easier. 
Consequently, instances of the protocol representing the out-of-phase scenario where D > 1 and 
d = 0, and hence, A aa > 1, have been explored. Thus far, the analyses indicate that the protocol 
solves the out-of-phase scenario. Instances of the protocol representing a more complex system 
where D > 1 and 0 < d < 1 have also been examined. Thus far, the analyses indicate that the 
protocol is applicable to realizable systems and practical applications. In addition, some 
instances of the protocol representing larger systems, where F > 1, have also been studied. Thus 
far, the analyses indicate that the protocol does not solve the general case of this problem 
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where F > 1. A detailed explanation of the analyses is beyond the scope of this report. 
Nevertheless, so far this model checking effort proved that, at a minimum, a deterministic 
solution for specific cases of this problem exists. We expect that this protocol serves as the 
starting point toward finding a comprehensive solution for the general case. In-depth analyses of 
the simplified version of this protocol for more complex and larger systems will be the subject of 
a subsequent report. This analysis will include pitfalls, relevant counterexamples, an argument 
toward impossibility results, as well as scenarios where this protocol can be used as a basis for 
larger systems and, thus, for realizable systems and practical applications. 
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Appendix A. Symbols 


The symbols used in the protocol are described in detail in [Malekpour 2006B] and are 
listed here for reference. 


Symbols 

P 

d 

D 

F 

G 

K 

K g 

Resync 

Affirm 

R 

A 

T a 

T r 

Restore 

Maintain 

T 

M 

P T.min 

Pt 

Pm 

P Effective 

Aaa 

Arr 

c 

AinculJimeA t ) 
Ap revision 

Aorift 

Ni 

Mi 


Descriptions 

bounded drift rate with respect to real time 

network imprecision 

event-response delay 

maximum number of faulty nodes 

minimum number of good nodes 

sum of all nodes 

set of all good nodes 

self-stabilization message 

self-stabilization message 

abbreviation for Resync message 

abbreviation for Affirm message 

threshold for Accept( ) function 

threshold for Retry( ) function 

self-stabilization state 

self-stabilization state 

abbreviation for Restore state 

abbreviation for Maintain state 

minimum duration while in the Restore state 

duration while in the Restore state 

duration while in the Maintain state 

the effective self-stabilization period 

time difference between the last consecutive Affirm messages 
time difference between the last consecutive Resync messages 
convergence time 

maximum time difference of Local_Timers of all good nodes at real time t 

self-stabilization precision 

maximum deviation from the initial synchrony 

the i th node 

the i th monitor of a node 
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